One approach to throttling requests made to network services involves specifying the maximum number of requests per second, or other time period, that are permitted to be made to the network service. Any requests that are received in excess of the specified maximum number of requests in a given time period are throttled (i.e. rejected). This mechanism is sometimes referred to as “static throttling.”
The static throttling mechanism described above suffers from several drawbacks. For example, the operator of a network service might not have control over all of the processes that are executed on the service host computer utilized to execute a network service. For instance, the operator might not have control over the periodic execution of maintenance tasks on the service host. These maintenance tasks consume central processing unit (“CPU”) cycles, memory, and, potentially, other resources that might otherwise be available for use by the network services.
A network service operator might, therefore, specify the maximum number of requests that are permitted to be made to a network service in a given time period pessimistically in view of the possibility that other processes might utilize CPU cycles, memory, or other resources of the service host, even though the other processes might only be executed periodically. As a result, the true capacity of a network service to process requests might not be realized.
It is with respect to these and other considerations that the disclosure made herein is presented.